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ABSTRACT 

Real  time  display  of  color  imagery  has  been  demonstrated  using  a  two- 
color  midwave  HgCdTe  IRFPA  built  by  Raytheon  SBRC.  A  variety  of  algorithms 
were  implemented  using  a  real  time  image  processor  (TI  C80)  on  a  PC  based 
frame  grabber  board.  Statistically  based  scaling,  de-correlation,  spectral  band 
differencing,  and  histogram  equalization  have  been  applied  using  the  red-cyan 
color  space  representation  of  the  imagery. 

1.0  INTRODUCTION 

Two-color  midwave  IRFPAs  are  being  developed  for  missile  approach  warning  sensors1, 
which  perform  target  identification  using  an  autonomous  processor.  The  high  degree  of 
correlation  between  300  K  thermal  signatures  for  the  two  closely  spaced  bands,  makes  the 
spectral  discrimination  of  spectrally  sharp  non-thermal,  (e.g.  C02  emission)  or  high  temperature, 
(e.g.  solar  reflection)  sources  possible  using  a  simple  weighted  difference  of  the  two  bands.  In 
spite  of  the  closeness  of  the  bands  however,  there  are  still  emissivity  and  reflectivity  related  color 
differences  for  nominally  graybody  objects,  which  can  form  the  basis  for  color  display  of  the 
information.  This  work  investigates  the  performance  of  two  color  midwave  focal  planes  as 
imaging  sensors  with  real  time  color  display  to  aid  in  visual  feature  identification.  Real  time 
display  of  color  imagery  formed  by  fusing  information  from  a  two-color  midwave  HgCdTe 
IRFPA  built  by  Raytheon  SBRC,  using  a  variety  of  algorithms  implemented  in  a  real  time 
processor  (TI  C80)  on  a  PC  based  frame  grabber  has  been  demonstrated  for  the  first  time. 
Statistically  based  scaling,  spectral  band  differencing,  and  histogram  equalization  have  been 
applied  using  the  red-cyan  color  space  representation  of  the  imagery. 

2.0  IRFPA  PERFORMANCE  CHARACTERIZATION 

The  two  color  midwave  focal  planes2  have  been  characterized  while  operating  at  a  60Hz 
frame  rate  in  a  50  mm  focal  length  f/2.3  camera  (SEIR  Inc)  at  80K.  The  focal  planes  have 
approximate  cutoff  wavelengths  of  4.0  and  4.5  microns,  and  exhibit  spectral  cross  talk  of  less 
than  5%  away  from  the  crossover  region  (Figure  1).  The  cut-on  wavelength  for  the  shorter  band 
was  defined  by  a  warm  dewar  window  and  was  3.3  microns.  The  mean  FPA  digital  outputs  for 
integration  times  of  2.0  and  3.0  msec  for  the  short  and  long  band  respectively,  are  shown  as  a 
function  of  blackbody  temperature  in  Figure  2.  Responsivities  of  51  and  76  counts/K  were 
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Figure  1  FPA  spectral  response 


Blackbody  temp(Kelvin) 

Figure  2  FPA  blackbody  responsivity 
for  longer  (red)  and  shorter  (blue)  bands. 


Figure  3  Responsivity  (left)  and  offset  (right)  maps  for 
MW1  (shorter)  band 


Measured  for  the  shorter  and 
longer  band  respectively. 
Combined  with  noise  levels  of 
1.4  and  1.7  counts  for  the  two 
bands,  NEDT’s  of  33  and  24 
mK  were  calculated  for  the 
respective  bands.  Responsivity 
uniformity  was  found  to  be 
5.5%  and  3.5%  (sigma/mean) 
for  the  two  bands.  These 
results  are  in  good  agreement 
with  earlier  characterization  of 
similar  arravs. 


Because  post  correction 
uniformity  plays  an  important 
role  in  determining  the  quality 
of  color  fused  imagery, 
extensive  characterization  of 
FPA  performance  was  carried 
out  in  this  area.  FPA  output 
was  offset  and  gain  corrected 
using  two  point  blackbody 
calibration  at  25  and  35  C. 
Maps  of  the  gain  and  offset 
coefficients  are  shown  in 


Figure  4  Responsivity  (left)  and  offset  (right)  maps  for 
MW2  (longer)  band 

Figures  3  and  4.  Responsivity  coefficients  show  a  near  random  distribution,  with  a  few  extended 
features.  The  two  dark  extended  features  evident  in  both  bands  correspond  to  faintly  visible 
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Figure  5  Responsivity  scatter  plot 
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Figure  6  Offset  scatter  plot 


defects  on  the  CdTe  side  of  the  FPA.  The  offset 
coefficients  show  a  cross  hatched  pattern  aligned 
with  the  FPA  rows  and  columns,  with 
superimposed  randomly  distributed  bright 
individual  pixels.  In  the  longer  band,  these 
occasionally  correspond  to  pixels  with  large  low 
frequency  noise,  or  blinkers.  No  evidence  for 
such  behavior  is  observed  for  the  shorter  band. 
Scatter  plots  of  the  responsivity  and  offsets  for 
each  band  are  shown  in  Figures  5  and  6.  Some 
degree  of  correlation  between  the  bands  for  both 
coefficients  is  clearly  evident,  suggesting  a 
common  origin.  The  alignment  and  symmetry  of 
the  cross  hatch  pattern  suggests  that  it  is 
dominated  by  some  readout  related  issue.  Post 
correction  spatial  uniformity  as  a  function  of 
background  temperature  for  calibration  points  of 
25  and  35  C  is  shown  in  Figure  7.  For  scene  temperatures  from  20  to  40  C,  the  sigma./mean  is  < 
.002,  or  approximately  twice  the  temporal  noise. 


Figure  7  Post  correction  spatial  noise  as 
a  function  of  background  temperature 


3.0  COLOR  PROCESSING  -  HARDWARE  IMPLEMENTATION 


The  color  display  algorithms  to  be  described  below  have  been  implemented  on  a  Matrox 
Genesis  PCI  bus  frame  grabber  card  in  a  200  MHz  Pentium  Pro  based  PC.  This  frame  grabber 
captures  the  nonstandard  format  12  bit  digital  output  from  the  camera  head  over  a  16  bit  parallel 
RS422  bus.  Because  all  image  processing,  as  well  as  image  display  functions,  are  performed  on 
the  frame  grabber  card,  neither  processing  power  or  bus  speed  of  the  host  machine  are  critical  in 
reaching  the  achieved  performance.  Image  display  is  via  a  24  bit  RGB  look  up  table  to  a 
dedicated  VGA  display.  This  signal  has  also  been  converted  to  RS170  video  format  for 
recording  on  standard  VHS  video  tape.  16  bit  integer  arithmetic  calculations  are  performed  using 


the  TI  C80  digital  signal  processing  chip  (single  processing  node  configuration)  on  the  card.  The 
processor  is  capable  of  up  to  2  billion  RISC  like  operations  per  second.  The  processor  also 
includes  a  100  MFLOP  floating  point  unit.  16  bit  integer  arithmetic  is  maintained  until  the  last 
step  in  the  display  process  to  maintain  the  full  dynamic  range  of  the  data,  without  clipping  or 
digital  wrap  around  effects.  Basic  arithmetic  array  functions  are  accessed  using  C  language 
library  routines  that  are  supplied  with  the  card.  Array  sums,  differences,  products,  and  logical 
operations  are  performed  real  time,  at  a  rate  consistent  with  60  Hz  frame  rates  and  a  128x128 
array.  The  frame  grabber  architecture  is  such  that  additional  processing  nodes  could  be  added  to 


Table  1  Processing  times  for  a  variety  of  functions 

Time 

16  bit  integer  addition  of  scalar  to  128x128  array 

<1.2  msec 

16  bit  integer  scalar  multiplication  of  128x128  array 

<1.2  msec 

16  bit  integer  scalar  division  for  128x128  array 

<2.7  msec 

16  bit  integer  subtraction  two  128x128  arrays 

<  1.2msec 

16  bit  integer  multiply(2  factors)  and  add  with  integer  arrays 

<1.2  msec 

16  bit  integer  look  up  table  mapping  128x128  array 

<1.4  msec 

3x  Zoom  of  128x128  array 

<2.2  msec 

work  in  parallel,  to  supply  the  throughput  required  for  larger  format  arrays.  No  low  level 
programming  of  the  C80  is  required  to  implement  the  desired  algorithms.  Processing  speeds 
achieved  for  a  variety  of  functions,  including  overhead  time  to  make  the  measurement,  is  shown 
in  Table  I. 


4.0  INFRARED  COLOR  MECHANISMS 

Infrared  photon  flux  emitted  by  a  surface  at  some  wavelength  is  a  function  of  its 
temperature  (as  defined  by  the  Planck  distribution  law)  and  its  surface  emissivity.  Natural 
variations  in  surface  temperature  and  emissivity  both  contribute  to  variations  in  the  overall 
photon  flux  emitted  from  any  surface.  Even  for  closely  spaced  bands,  these  variations  can 
provide  the  basis  for  color  imagery.  During  daytime  imaging  in  the  midwave  bands,  there  is  also 
a  significant  contribution  to  the  scene  from  reflected  solar  radiation.  This  becomes  an  additional 
source  of  color  as  a  result  of  reflectivity  differences,  and  the  very  different  spectral  distribution  of 
the  6000K  illumination,  from  the  near  300K  self  emitted  radiation.  When  imaging  a  target  at 
ambient  temperature  in  background  clutter  with  a  single  band  infrared  sensor,  the  target  can 
appear  to  have  either  a  positive  or  negative  contrast.  Furthermore,  using  a  monochrome  display, 
there  is  the  additional  limitation  that  a  human  viewer  has  an  instantaneous  dynamic  range 
equivalent  to  only  100  shades-of-gray  (a  one  percent  contrast  difference).  Therefore,  for  scenes 
with  high  background  clutter,  targets  can  be  hard  to  discriminate  because  of  confusing  contrast 
inversion  possibilities,  or  the  probability  that  the  target  and  the  local  background  could  have  zero 
contrast. 

The  goal  of  two  band  IR  color  fusion  is  to  overcome  the  contrast  limitations  that  exist 
with  single  band  imagery.3  By  using  two  infrared  bands  to  create  a  color  space,  an  associated 
composite  color  image  can  be  created  for  display  to  human  viewers.  In  so  doing,  the  likelihood  of 
detection  is  increased  because  either  the  target’s  brightness  or  color  can  be  different  from  the 
background.  In  this  regard,  discriminating  targets  from  backgrounds  using  IR  color  fusion  is  very 


similar  to  color-and-shape  discrimination  in  human  color  vision.  The  power  of  this  approach  is 
that  it  leverages  the  known  human  ability  to  discriminate  millions  of  colors  defined  by  varying 
hue,  saturation,  and  brightness. 

5.0  COLOR  DISPLAY  ALGORITHMS 

.Because  the  human  color  vision  system  has  three  types  of  photoreceptors  (namely  red, 
green  and  blue  cones),  viewing  two  band  imagery  is  a  not  a  perfect  fit.  The  human  color 
processing  system  works  with  color  opponents  (namely  red-green  and  yellow-blue  color 
opponency).  With  this  understanding,  it  is  possible  to  use  a  variant  of  this  to  display  a  two  band 

infrared  image.  In  the  work  done  at  NRL,  a  red- 
green  color  opponency  was  chosen.  In  practice, 
the  longer  spectral  wavelength  infrared  band  was 
input  to  a  red  display  channel  and  shorter 
wavelength  band  was  input  to  both  the  green  and 
the  blue  display  channels.  The  combination  of 
equal  amounts  of  green  and  blue  creates  cyan.  In 
essence  this  is  the  same  as  what  is  known  as 
human  red-green  color  opponency.  Red  and  cyan 
are  true  color  opponents  because  when  they  are 
combined  in  equal  quantities,  they  create  the 
perception  of  white,  black,  or  some  shade  of 
gray.  In  this  manner  the  relative  intensities 
between  the  two  bands  at  each  pixel  can  be 
represented  as  a  chromatic  continuum  starting  as 
red,  going  through  gray,  and  ending  as  cyan.  At 
the  same  time  a  brightness  value  can  be  assigned 
in  an  orthogonal  direction  (Figure  8).  Therefore, 
each  pixel  has  a  chrominant  value  (red-cyan)  and 
a  brightness  value  (black-white).  It  is  interesting 
to  note  that  if  the  data  has  a  large  dynamic  range  in  either  the  brightness  or  chrominant 
directions,  then  a  nonlinear  compression  technique  (such  as  histogram  equalization)  can  be 
employed  in  that  coordinate  system  without  a  major  distortion  of  color  values  as  might  occur 
when  performing  compression  in  the  original  coordinate  system. 

Two  important  issues  arise  when  processing  two  band  infrared  imagery  to  create  good 
composite  color  imagery.  The  first  is  that  of  obtaining  good  color  contrast  between  objects  and 
backgrounds.  The  second  is  obtaining  good  color  constancy  regardless  of  illumination  and 
temperature  -  that  is,  target  and  background  colors  should  be  the  same  at  night  as  during  the  day, 
or  the  same  regardless  of  whether  the  object  is  hot  or  cold.  The  brightness  or  the  darkness  of  the 
object  may  vary,  but  the  color  (hue  and  saturation)  should  remain  the  same.  In  this  paper  only 
color  contrast  enhancement  will  be  discussed.  The  issue  of  color  constancy  is  more  complex  and 
will  be  the  focus  of  future  real  time  imaging  demonstrations.4 

In  summary,  the  approach  to  color  fusion  is  to  map  the  raw  image  data  into  an  appropriate 
color  space,  and  then  processing  it  to  achieve  an  intuitively  meaningful  color  display  for  a  human 
viewer.  The  resulting  imagery  should  provide  good  color  contrast  between  objects  and 


MW2  (RED) 


Figure  8  Red-cyan  color  space.  Brightness 
direction  shown  green  arrow,  chromaticity 
by  black  arrow. 


backgrounds  and  consistent  colors  regardless  of  environmental  conditions  such  as  solar 
illumination  and  variations  in  surface  temperature.  The  algorithm  effort  at  NRL  has  focused  on 
attempting  to  first  understand  the  underlying  infrared  phenomenology  and  then  apply  traditional, 
linear  color  processing  techniques  as  appropriate.  After  this  approach  is  fully  explored  it  may  be 
useful  to  expand  into  the  area  of  nonlinear  processing.  But  for  the  present  there  remains  a  great 
deal  to  understand. 

The  fundamental  color  mechanisms  are  exhibited  in  the  following  laboratory  data,  where 
straightforward  assignments  of  the  longer  wavelength  and  shorter  wavelength  data  to  the  red  and 
cyan  color  planes  are  made.  In  this  process,  linear  scaling,  based  on  the  statistical  properties  of 
the  data  for  each  band,  is  applied  to  each  band  to  normalize  the  data  to  have  common  means,  and 
maxima.  The  transmission  spectra  for  a  sheet  of  polystyrene,  (round  object  in  the  image)  and 
teflon  , (square  object  in  the  image)  each  approximately  1  mm  thick  are  shown  in  Figure  9  The 

teflon  can  be  seen  to  be  more  transparent  in 
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Figure  9  Transmission  of  polystyrene  (red) 
and  teflon  (cyan) 


the  shorter  wave  band,  and  less  transparent 
in  the  long  wavelength  band,  with  the 
polystyrene  behaving  just  the  opposite. 
Figure  10  shows  fused  color  imagery 
including  the  two  objects.  When  backlight 
with  a  blackbody  source,  the  teflon  appears 
as  cyan,  while  the  polystyrene  appears  as  red. 
When  the  objects  are  heated  slightly  above 
ambient  and  held  in  front  of  a  dark 
background,  then  the  signature  is  dominated 
by  emission,  and  the  teflon  appears  red  and 
the  polystyrene  appears  cyan.  A  signature 
which  is  a  combination  of  transmission  and 
emission  characteristic  can  be  observed 
where  the  warm  objects  are  backlight  with  a 
warm  blackbody  source  (finger)  as  well. 


Figure  10  Polystyrene  (round)  and  teflon  (square)  sheets  in  transmission  and  emission 


.One  problem  that  typically  arises  when  mapping  infrared  data  into  a  color  space  is  that 
the  corresponding  pixel  values  between  two  bands  are  often  highly  correlated  for  those  infrared 

wavelengths  that  are  predominantly  self- 
emitting.  This  is  particularly  true  of  any  sub¬ 
bands  in  the  MWIR  and  LWIR  spectral 
bands  at  nighttime.  Note  that  other  infrared 
regions  such  as  the  SWIR  and  NIR  spectral 
bands  are  predominately  used  in  daytime 
imaging  systems  that  view  solar-reflected 
photons.  The  correlation  between  two 
MWIR  bands  for  a  night  scene  is  shown  in 
Figure  11.  The  distribution  of  pixel  values 
tends  to  be  a  locus  of  points  that  runs 
diagonally  through  the  scatter  plot  of  the 
original  color  space.  Simple  scaling  or 
normalizing  of  each  band  individually 
results  in  the  image  in  Figure  12,  which 
displays  relatively  little  color  content  other 


Figure  11  Scatter  plot  for  correlated  scene 
shown  below 


than  the  C02  emission  related  signatures  from  the  smokestacks  from  the  power  generating  plant. 

.Color  contrast  enhancement  can  be 


achieved  using  a  two-step  process.  First,  a  new 

coordinate  system  is  established  where  the 

primary  axis  is  in  the  brightness  direction  and  a 

secondary,  orthogonal  axis  in  the  chrominant 

direction.  The  primary  axis  (or  principal 

component  direction )  is  based  on  the  pixel 

statistics.  In  the  most  general  formulation,  this  is 

computed  by  finding  the  covariance  matrix  of  all 

the  pixel  values  and  then  finding  the  eigenvectors 

of  the  covariance  matrix.  Using  the  eigenvectors 

of  the  largest  eigenvalue  (which  relate  to  the 

principal  component  direction  for  correlated 

data)  a  rotation  matrix  can  be  found  that 

translates  the  values  from  the  original 

coordinates  into  the  new  coordinate  system.  The 

^  „  , ,  principal  component  direction  is  taken  to  be  the 

Figure  12  Fused  color  image  formed  by  u  •  u.  ,•  ,  .,  u  .... 

&  n  ,  ,  ,  brightness  direction,  and  the  chromaticity 

linear  scaling  of  each  band  ..  ..  ,  .  .  ,  . 

&  direction  is  orthogonal  to  it.  The  new  coordinate 

system  (with  one  axis  being  a  principle  component  direction)  is  very  useful  because  it  allows 

certain  types  of  color  processing  to  be  performed  in  a  separable  manner.  For  example,  because 

the  two  bands  are  highly  correlated,  the  pixel  values  tend  to  lie  near  the  principal  coordinate  axis, 

whereas  if  the  data  is  uncorrelated  it  will  tend  to  have  a  circular  distribution.  Correlated  data  can 

be  de-correlated  in  the  transform  space  to  achieve  color  contrast  enhancement.  That  is,  the 

luminance  and  chromimance  values  data  can  be  re-scaled  separately  in  the  new  coordinate 

system,  then  rotated  back  into  the  original  coordinate  system,  resulting  in  decorrelation  of  the 
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Figure  13  Scatter  plot  after  decorrelation 
for  image  shown  adjacent. 


Figure  14  fused  color  image  after 
decorrelation 


original  data.  In  practice,  the  principal  component  direction  has  been  assumed  to  be  at  45 
degrees  to  the  red  and  cyan  axes.  In  simplified  approximation,  the  brightness  is  then  the  sum  of 
the  short  and  long  band  pixel  values,  and  the  chrominance  is  the  difference  of  the  short  and  long 
band  values.  Color  enhancement  is  achieved  by  multiplicative  scaling  of  the  chrominance  data, 
followed  by  transformation  back  to  the  red-cyan  color  space.  In  the  inverse  transformation,  red  is 
the  difference  of  the  brightness  and  color,  and  cyan  is  the  sum  of  the  brightness  and  color.  An 
example  of  decorrelated  image  data  is  shown  in  Figure  13  and  14. 

Under  solar  illumination,  the  principal  component  axis  no  longer  lies  equidistant  between 
the  red  and  cyan  axes  because  of  the  lack  of  correlation  between  the  two  bands.  The  algorithm 
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Figure  15  Scatter  plot  for  strongly  Figure  16  Fused  color  image  with  strong 

uncorrelated  image  dominated  by  solar  solar  reflection  component 

reflections 


Figure  17a  &  b  Highly  correlated  image  with  weak  MW2 
signal  before  and  after  difference/threshold  process 


described  above  produces  unpredictable  color  balance  under  those  conditions.  Because  the 
correlation  between  the  bands  is  weak  however,  significant  color  contrast  can  be  achieved  using 
simple  scaling  of  the  information  from  the  two  bands.  Such  an  image  is  shown  along  with  its 
scatter  plot  in  Figures  15  and  16.  Solar  reflections  are  clearly  indicated  by  the  strong  cyan 
components  visible  in  the  image. 

.In  a  heavily  correlated  scene,  where  some  pixels  which  lie  just  beyond  the  edge  of  the 
distribution  for  the  rest  of  the  scene,  it  is  possible  to  identify  those  pixels  by  applying  a 

thresholding  function  to  a 
difference  of  the  two  bands. 
This  identification  can  then  be 
used  to  key  these  pixels  to  a 
unique  color  with  large 
contrast  to  the  rest  of  the 
scene.  This  can  occur  when  a 
relatively  dim  long  band 
source,  e.g.  low  level  C02 
emission  from  a  butane  lighter 
is  back-light  with  a  hot 
blackbody  source  (Figure 
17a).  Strong  color  contrast 
may  be  regained  by  using  the 
difference  information  as  a  key  to  turn  off  the  green  and  blue  components  of  a  pixel  which 
exceeds  the  threshold  (Figure  17b).  Although  this  approach  does  not  result  in  a  ‘realistic’  color 
image,  it  provides  a  clear  visual  cue  to  a  signature  of  interest. 

Figure  18  shows  an  image  obtained  by  a  uniform 
histogram  equalization  of  each  band  independently  prior 
to  assignment  of  colors.  The  image  has  the  advantage  of 
displaying  the  full  dynamic  range  of  the  scene  with  strong 
color  contrast.  However,  it  has  the  disadvantage 
discussed  earlier  of  resulting  in  wide  variations  in  color 
from  scene  to  scene,  depending  on  what  reference  is  used 
for  the  equalization.  Future  plans  include  using  this 
approach  in  the  chrominance-brightness  space  to  achieve 
improved  color  contrast,  with  good  constancy. 

Because  the  de-correlation  routine  involves  the 
sums  and  differences  of  the  two  bands,  both  temporal  and 
fixed  pattern  noise  is  magnified.  This  problem  is 
magnified  by  the  fact  that  the  spatial  noise  for  the  two 
bands  is  correlated  to  some  degree.  Fixed  pattern  noise 
typically  limits  the  extent  to  which  the  color  can  be 
enhanced  without  degrading  the  quality  of  the  image.  An 
example  of  very  high  brightness  and  high  color  gain  image  is  shown  in  Figure  19.  The  observed 
noise  is  nearly  all  the  result  of  fixed  pattern  effects  rather  than  temporal  noise. 


Figure  18  Histogram  equalization 
per  band  image. 


6.0  SUMMARY 


Figure  19  Fixed  pattern  noise  under 
high  contrast  and  color  gain 
conditions. 


Real  time  implementation  of  a  variety  of  color 
fusion  algorithms  has  been  demonstrated  with  a 
simultaneously  integrating  co-located  two  color 
midwave  IRFPA.  Along  with  providing  visual  cues  for 
spectral  sharp  signatures  associated  with  C02  emission, 
good  color  contrast  has  been  achieved  for  both  night  and 
daytime  imagery.  While  sensor  NET  is  degraded  over 
either  single  band  NET,  by  amplification  of  both 
temporal  and  fixed  pattern  noise  in  the  color  generation 
process,  the  additional  information  provided  by  color 
compensates  in  part  for  this  in  target  identification. 
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